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26 November 1968 


MEMORANDUM FOR: Mr, Ba 


SUBJECT : Summary of Two AES Reséarch Reports on CT's 


1. The first report finds that there is no relationship between 
training evaluations (oe Support Services Course and Manpower Ratings 
of job performance and potential, This finding is rather academic because 
training evaluations are no longer given in the Support Services Leader 


2. The second report finds that for the 20 CT classes between 
September 1959 and February 1967 there is really no difference in the 
ability between those CT's who drop out quickly and those who stay. It 
also notes that CT's who leave after assignment to Agency positions are 
a little brighter than those who stay. However the difference in ability 
is "only modest" and the C’'I’s who stay are in the top half of Agency pro- 
fessionals and the top 5% of general population in intellectual ability. 


3. Idon't see that these findings suggest any practical 
guideline for improved management of the CT Program. They do suggest 
however the need for further research into the sources of job dissatisfaction 

and reasons for leaving the Agency. 
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1h November 196€ 


MEMORANDUM FOR: Deputy Director for Support 
THROUGH : Chief, Assessment and Evaluation Staff, OMS 


SUBJECT : Elaboration of Research Findings Reported in 
Systems Analysis of Psychological Data 
Progress Report 


REFERENCE : Periodic Progress Report of 22 October 1963 


1. Attached are the two AES research reports you requested. The 
first deals with the finding of no relationship between training evalua- 
tions in the Support Services Course and the Manpower Ratinegs of job 
performance and potential. ‘The second compares the intellectual abilities 
of CIs who left the Agency with those who remained. 


2. We would like to express our appreciation for the opportunity 
which you recently provided us to brief your staff. 


25X1 


25X11 


Acting Chief, Research Branch 
Assessment and Svaluation Staff 
Office of Medical Services 


Acting Chairman, Task Force on Systems Analysis 
of Psychological Data Pertaining to Career Trainees 


Attachments: 
Ae Discussion of Finding of No Relationships between 
Support Services Course Training Evaluations and 
Ratings of Job Performance and Potential 
B. Comparisons of Intellectual Abilities of Career Trainees 
Who Left the Agency with Those Who Remained 


Distribution: 
Orig. and 1 - DDS 
1 - D/MS 
1 - C/AES 
OMS/AES/ GYD :mak 
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ATTACHMENT A 


DISCUSSION OF FINDING OF NO RELATIONSHIPS BETWEEN 
SUPPORT SERVICES COURSE TRAINING EVALUATIONS AND 


RATINGS OF JOB PERFORMANCE AND POTENTIAL 
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This memo responds to your request for elaboration of the research 
finding that training evaluations in the Support Services Course (S.S.C.) 
were not predictive of ratings of job performance and potential. Herein 
we will describe more fully than was possible in our earlier briefing the 
details and findings of the research relating 5.5.C. and other CT training 
course outcomes to the ratings collected by the Committee on Professional 
Manpower. 

Basically, what we found was that for a sample of 93 male CTs, the 
relationships between final course evaluations in S.S.C. and Manpower 
ratings were essentially random ones. Conversely, for all other training 
courses taken by CTs,* statistically significant relationships were found 
between training grades and one or more of the ratings of performance and 
potential. We hasten to add that the relationships involving these other 
courses, while often significant, are not in absolute terms overly 
impressive. Typically, only a small proportion of the variability of 
job ratings is predictable from final training grades alone. The 
significance of the discrepant pattern of results for S.S.C. should be 
assessed in light of certain features of the course itself, the sample 
involved in the analyses, the grading system, etc. This background 


information is given below: 
* 
(Excluding OFC, for which insufficient data for analyses were available) 
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Course Objectives -- S.8.C. (Abstracted from course description): 


The Support Services Course is designed for Career Trainees 
who will be assigned to the Support Services. The progran 
intends to orient the Career Trainee to the mission and 
functions of the Support Services. Indoctrination in some 
detail is given in the various aspects of support pertinent 
to the operation of a field station. Included in this 
instruction is coverage of personnel, security, medical, 
finance, logistics, and travel subjects. 


Sample : 


The sample on which data analyses were based consisted of the 93 
male CTs who had taken the S.S.C. sometime between 25 January 1965 
(S.8.C. #1) and 1 December 1967 (S.S.C. #1-68) and on whom Men- 
power ratings were available. It should be noted that this 

group represents less than one-half of the CIs who have taken 

the S.S.C. since its inception to date. We are unaware of any 
selective biases differentiating this group from the remaining 

CIs who have taken 5.5.C. 


Achievement Record: 


With but one minor modification, the basis for assigning grades 

in S.S.C. remained unchanged over the seven course runnings from 
which the study sample was drawn. Students received grades on 

(1) a "First Week Test" covering organization and functions of 

the Headquarters elements and CIA support policies; (2) a "Logistics 
Test"; (3) a "Finance Test" including preparation of a comprehensive 
travel voucher -- in earlier runnings of the S.5.C., a separate 
"Travel Test" was administered; and (4) a "Comprehensive Problem" 
consisting of a series of practical exercises. 


The overall grade given in S.S.C. is a weighted summation of the 
grades received on the above elements. This summary numerical 

grade is converted to and reported in terms of the adjectival 
ratings used in Fitness Report rating -- Weak (W), Adequate (A), 
Proficient (P), Strong (S), and Outstanding (0). (In fact, grades 
of P, S, and O were the only ones assigned to the CTs in the sample. ) 


It 
In the following section, we develop a number of hypotheses to account 


for the finding of no systematic relationship between 5.5.C. grades and job 
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vatings. Where the availability of data permitted tests of these 
hypotheses, results are reported. 

Statistically significant relationships are difficult or impossible 
to obtain when the distribution of scores on either the predictor measure, 
in this case final course evaluations in S.5.C., or on the criteria, Man- 
power ratings, is severely restricted. In light of this statistical 
requirement, we hypothesized that either or both of these distributions. 
were restricted for the S.S.C. sample. We found in fact that the spread 
of scores on the Manpower ratings of overall performance and potentiel for 
the total CT Manpower sample {N=386) and for the S.S.C. sample did not 
differ significantly. We also found no reliable differences in average 
performance and potential ratings of these two samples. 

Where comparability of grading systems permitted comparisons with other 
training courses, we found that the distribution of grades in 5.5.C. tended 
to be only moderately more restricted than those in other courses, and to a 
statistically significant degree in only two of four instances. Hence, it 
is difficult to attribute the failure to find relationships between 5.5.C. 
and job ratings to the distribution of predictor and criterion measures 
alone. 

Since the S.S.C. has been given only since January 1965 and since 
Manpower ratings were collected in January 1968, we expected and found 
that one-third of the S.S.C. sample had been on the job less than one 


year at the time job ratings were collected. We hypothesized that ratings 
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made of people on the job for such a short period of time might be highly 
unreliable and biased, and that by including them in the analyses, true 
relationships between course and job performance would be obscured. 
Accordingly, we re-analyzed our data, restricting the study sample to 
those CIs (N=59) who had been on the job for at least one year. Again, 
we found no significant relationships between 5.S5.C. training evaluations 
and overall job performance and potential. 

Another explanation for the unique S.S.C.-Manpower results might be 
that major shifts had occurred in the assignment of grades over the several 
runnings of the Support Course. As noted above, the basis for evalueting 
students had not materially changed in this time, but this, of course, did 
not preclude shifts in average grade assigned. If this average grade 
fluctuated from class to class without a corresponding and real change in 
average class performance, differences in grading standards are indicated 
-- perhaps administratively imposed. The resulting discrepancies in the 
meaning of Proficient, Strong, and Outstanding performances over time would 
seriously reduce chances of obtaining meaningful relationships with external 
criteriae An approximate test for such shifts in grading standards is 
provided by comparing the average grades assigned in the different runnings 
of S.S.C. Our working assumption would be that these averages would not 
differ reliably. Based on data for 151 CIs in seven consecutive runnings 
of S.S.C. == January 1965 through November 1967 -- we found small but 


statistically significant differences in the average grade of classes. For 
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the first five classes, the average grade assigned increased progressively; 
this trend was reversed for the following two classes. This is not unusual 
for a newly developed course as criteria of training performance become 
refined and consensually validated only over time. 

We conelude this section by noting that the reported emphasis in 5.S.C. 
was on operating field stations and that the graded tests and exercises 
reflected this emphasis. If in fact many or most of the CTs included in 
the study sample were not performing in support roles in field stations at 
the time ratings were collected, the likelihood of finding relationships 
between training and job criteria would certainly be diminished. Unfortunately, 


we have no data to enlighten us on this possibility. 


TIL 

In this final section, we comment briefly on the assumption implicit 
in the interest shown in the lack of relationship between 5.5.C. and job 
ratings. To wit, positive and significant relationships should obtain 
between training and job criteria. 

CT training courses, especially those designed for CTs entering a 
specific directorate -- for example, IPC, SSC, OC -- prepare trainees to 
fill certain roles. They do this by teaching the skills and developing 
the knowledge and attitudes necessary to perform in those roles. To the 
degree that the demands of training mirror those of the job, systematic 
relationships between training and job ratings would be expected; however, 
without full appreciation of these demand characteristics, our expectancies 


become arbitrary. 
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Judging from the fact that performance in S.S.C. is significantly 
related to AES tests of non-verbal reasoning, mathematical abilities, and 
interests in accountancy, it would seem that the stated course emphases -- 
accounting, finance, and logistics -- are real ones. Competence in these 
applied areas is no doubt necessary for satisfactory job performance, but 
is not in itself a sufficient determiner of that performance. Evidently 
the more important predictors of job ratings lie in areas not covered or 
evaluated in the S.5.C. 

Finally, it will be remembered that all data analyses involving the 
Support Services Course were based on early runnings of the course. If 
substantive changes in teaching methods, content, grading, etc. have 
occurred over the past two years, the findings reported in this memo 


would be open to thorough review. 
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ATTACHMENT B 


COMPARISONS OF INTELLECTUAL ABILITIES OF 
CAREER TRAINEES WHO LEFT THE AGENCY WITH 


THOSE WHO REMATNED 
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